Method and apparatus for user interface of input devices

ABSTRACT

A system for a 3 dimensional (3-D) user interface comprises: one or more 3-D projectors configured to display an image at a first location in a 3-D coordinate system; one or more sensors configured to sense user interaction with the image and to provide user interaction information; and a processor configured (i) to receive the user interaction information from the one or more sensors; (ii) to correlate the user interaction with the image; and (iii) to provide one or more indications responsive to a correlation of the user interaction with the image, wherein the one or more indications comprise displaying the image at a second location in the 3-D coordinate system. A method for providing a 3-D user interface comprises: generating an image at a first location in a 3-D coordinate system; sensing user interaction with the image; correlating the user interaction with the image; and providing one or more indications responsive to a correlation of the user interaction with the image, wherein the one or more indications comprise displaying the image at a second location in the 3-D coordinate system. Computer readable program codes related to the system and the method of the present invention are also described herein.

BACKGROUND OF THE INVENTION

A graphical user interface (GUI) is a type of computer application userinterface that allows people to interact with a computer andcomputer-controlled devices. A GUI typically employs graphical icons,visual indicators or special graphical elements, along with text, labelsor text navigation to represent the information and actions available toa user. The actions are usually performed through direct manipulation ofthe graphical elements.

Holographic images can be created as single or consecutive images usingavailable holographic technology. These technologies include mirrors,lasers, light and images strategically positioned to cause the properreflection to yield a holographic image broadcast through an entry pointin the laser and mirror positioning system. Black background and roomswith low or no light may enhance the appearance of the holographic imageor images, which may also use a holographic plate as a display medium.Holographic systems may be large in size and spread out over a largebroadcasting area or may be compact enough to fit in spaces smaller thana desk top. Holographic technology is only limited in size by the sizeof the component parts. By using holographic technology, images may bedisplayed multi-dimensionally, rather simply on a planar projection.

Currently progress has been made in technologies that can enhance thecapability and range of holographic media in projects that employmulti-million mirror systems and via companies that have designedspecialized high speed and high capacity micro processors forspecialized jobs, other than holographic systems, where the technologycould be applied to holographic technologies to make possible the properpositioning of millions of mirrors at a rate of between 24 to 60 or moreframes of video per second, with corresponding synched audio.

Holographic displays generated over the last 20-year period utilizevarious configurations including lasers with images on glass plates suchas an AGFA 8E75HD glass plate or other glass plates as well a laser suchas a Spectra Physics 124B HeNe laser, a 35 mW laser diode systemutilizing different processing methods such as pyrochrome processing.Split beam techniques can also be used Multi H1 to Multi H2. Suchconfigurations as 8×10, triethanolomine, from Linotronic 300 imagesetter film are also commonly utilized or a configuration withrear-illuminated for 30×40 cm reflection hologram, where a logo floats18-inches in front of the plate.

SUMMARY OF THE INVENTION

Some user interfaces have adopted a multi-dimensional interfaceapproach. For example, the “heliodisplay” of IO2 Technology, LLC of SanFrancisco, Calif. projects images into a volume of free space, i.e. intoan aerosol mixture such as fog or a gas, and may operate as floatingtouchscreen when connected to a PC by a USB cable. However, with theheliodisplay, the image is displayed into two-dimensional space (i.e.planar). While the Heliodisplay images appear 3 dimensional (“3-D”), theimages are planar and have no physical depth reference.

Unfortunately, these existing uses have certain limitations indistribution and deployment. For example, functionally, the heliodisplayis a two dimensional display that projects against a curtain of air, oreven glass. While, the heliodisplay may give the appearance of 3-D, theimages displayed and the interface are 2-D. As such, the heliodisplay isnot a true 3-D holographic display, and thus the interface operates on atwo-dimensional plane, not taking advantage of a full three dimensionalcoordinate system.

Accordingly, there is a need for an integrated user interface thatutilizes true 3-D technology to create a computing and multimediaenvironment where a user can easily navigate by touch, mouse or pointersystem to effectively navigate the interface to raise the level of theuser experience to a true 3-D environment, with the goal of attainingelements of the attenuated clarity, realism and benefits of thatenvironment that match our day to day conventional interactions with the3-D world. The present invention relates to the creation of aholographic user interface display system that combines physical mediaor digitally stored files with a digital holographic player hardwaresystem. The result is the creation of a multimedia holographic userinterface and viewing experience, where a variety of graphicalschematics enabling cohesive access to information utilizing pyramids,blocks, spheres, cylinders, other graphical representations, existingtemplates, specific object rendering, free form association, userdelegated images and quantum representations of information to form auser interface where the available tools combine over time to match ausers evolving data and requests.

According to one aspect of the present invention, a system for a 3dimensional (3-D) user interface comprises: one or more 3-D projectorsconfigured to display an image at a first location in a 3-D coordinatesystem; one or more sensors configured to sense user interaction withthe image and to provide user interaction information; and a processorconfigured (i) to receive the user interaction information from the oneor more sensors; (ii) to correlate the user interaction with the image;and (iii) to provide one or more indications responsive to a correlationof the user interaction with the image. The one or more indications cancomprise displaying the image at a second location in the 3-D coordinatesystem. The system can further comprise a communications port coupled tothe processor and configured to provide communications interface with acomputer system. The image can be a holograph. The user interaction withthe image can comprise movement of the user relative to the image, orany sound produced by the user, or both. The one or more sensors cancomprise any one or combination of location sensors, motion sensors,light sensors and sound sensors. The location sensors can comprise lasersensors configured to geometrically identify a position within the 3-Dcoordinate system. The light sensors can comprise any one or combinationof photovoltaic sensors, image sensors and photo-electric light sensors.The sound sensors can comprise microphones. The image at the firstlocation can cease to be displayed before, when or after the image atthe second location starts to be displayed. The distance between thefirst location and the second location are within a range such that asequential display of the image at the first location and the secondlocation can be thought of as a movement of the image from the firstlocation to the second location. The user interaction can comprisemovement of the whole body or one or more body parts of the user fromthe first location to the second location, thereby creating an effect ofthe image moving from the first location to the second locationresponsive to the movement of the whole body or the one or more bodyparts of the user from the first location to the second location. Theimage can be displayed at the second location immediately after thewhole body or the one or more body parts of the user moves to the secondlocation, thereby creating an effect of the image moving with the wholebody or the one or more body parts of the user. The image can be that ofan input device for a computer, and the one or more indications canfurther comprise inputting data to the computer responsive to thecorrelation of the user interaction with the image. The data input tothe computer can comprise data that are related or not related to thespatial difference between the first location and the second location inthe 3-D coordinate system, or both. The input device can be selectedfrom the group consisting of: mice, light guns, steering wheels, yokes,joysticks, directional pads, wired gloves, game controllers, game pads,game paddles and Wii remotes. The input device can be a mouse and thesystem can further comprise an image of a keyboard in the 3-D coordinatesystem or an actual keyboard.

According to another aspect of the present invention, a method forproviding a 3 dimensional (3-D) user interface comprises: generating animage at a first location in a 3-D coordinate system; sensing userinteraction with the image; correlating the user interaction with theimage; and providing one or more indications responsive to a correlationof the user interaction with the image. The one or more indications cancomprise displaying the image at a second location in the 3-D coordinatesystem. The image can be a holograph. The user interaction with theimage can comprise movement of the user relative to the image, or anysound produced by the user, or both. Sensing can comprise using one ormore laser sensors to geometrically identify a position within the 3-Dcoordinate system. Sensing can also comprise using a sound sensor toidentify an audio input from the user. The image at the first locationcan cease to be displayed before, when or after the image at the secondlocation starts to be displayed. The distance between the first locationand the second location are within a range such that a sequentialdisplay of the image at the first location and the second location canbe thought of as a movement of the image from the first location to thesecond location. The user interaction can comprise movement of the wholebody or one or more body parts of the user from the first location tothe second location, thereby creating an effect of the image moving fromthe first location to the second location responsive to the movement ofthe whole body or the one or more body parts of the user from the firstlocation to the second location. The image can be displayed at thesecond location immediately after the whole body or the one or more bodyparts of the user moves to the second location, thereby creating aneffect of the image moving with the whole body or the one or more bodyparts of the user. The image can be that of an input device for acomputer, and the one or more indications can further comprise inputtingdata to the computer responsive to the correlation of the userinteraction with the image. The data input to the computer can comprisedata that are related or not related to the spatial difference betweenthe first location and the second location in the 3-D coordinate system,or both. The input device can be selected from the group consisting of:mice, light guns, steering wheels, yokes, joysticks, directional pads,wired gloves, game controllers, game pads, game paddles and Wii remotes.The input device can be a mouse and the method can further comprisedisplaying an image of a keyboard in the 3-D coordinate system orproviding an actual keyboard.

The method can further comprise providing at least one of the providedindications to a device on a network.

According to still another aspect of the present invention, a computerreadable medium has computer readable program codes embodied therein forcausing a computer to provide a 3 dimensional (3-D) user interface. Thecomputer readable program codes can cause the computer to: generate animage at a first location in a 3-D coordinate system; sense userinteraction with the image; correlate the user interaction with theimage; and provide one or more indications responsive to a correlationof the user interaction with the image. The one or more indications cancomprise generating the image at a second location in the 3-D coordinatesystem. The image can be a holograph. The user interaction with theimage can comprise movement of the user relative to the image, or anysound produced by the user, or both. Sensing can comprise using one ormore laser sensors to geometrically identify a position within the 3-Dcoordinate system. Sensing can also comprise using a sound sensor toidentify an audio input from the user. The image at the first locationcan cease to be displayed before, when or after the image at the secondlocation starts to be displayed. The distance between the first locationand the second location are within a range such that a sequentialdisplay of the image at the first location and the second location canbe thought of as a movement of the image from the first location to thesecond location. The user interaction can comprise movement of the wholebody or one or more body parts of the user from the first location tothe second location, thereby creating an effect of the image moving fromthe first location to the second location responsive to the movement ofthe whole body or the one or more body parts of the user from the firstlocation to the second location. The image can be displayed at thesecond location immediately after the whole body or the one or more bodyparts of the user moves to the second location, thereby creating aneffect of the image moving with the whole body or the one or more bodyparts of the user. The image can be that of an input device for anelectronic device, and the one or more indications can further compriseinputting data to the electronic device responsive to the correlation ofthe user interaction with the image. The data input to the electronicdevice can comprise data that are related or not related to the spatialdifference between the first location and the second location in the 3-Dcoordinate system, or both. The input device can be selected from thegroup consisting of: mice, light guns, steering wheels, yokes,joysticks, directional pads, wired gloves, game controllers, game pads,game paddles and Wii remotes. The input device can be a mouse and thecomputer readable program codes can further cause the computer togenerate an image of a keyboard in the 3-D coordinate system or toreceive input from an actual keyboard. The computer readable programcodes can further cause the computer to provide at least one of theprovided indications to a device on a network.

Embodiments of the invention provide a holographic user interface whichtransforms the computing environment to enable a three dimensionalholographic style user interface and display system. The system utilizesholographic projection technology along with programmed quadrantmatrixes sensor field to create multiple methods to select and interactwith data and user interface tools and icons presented in a holographicformat. The system may be used for interconnecting or communicatingbetween two or more components connected to an interconnection medium(e.g., a bus) within a single computer or digital data processingsystem.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing will be apparent from the following more particulardescription of example embodiments of the invention, as illustrated inthe accompanying drawings in which like reference characters refer tothe same parts throughout the different views. The drawings are notnecessarily to scale, emphasis instead being placed upon illustratingembodiments of the present invention.

FIG. 1 is a block diagram illustrating a holographic user interfaceaccording to an embodiment of the present invention;

FIG. 2 is a flow chart diagram illustrating a method for providing a 3dimensional (3-D) interface with a system;

FIG. 3 is a perspective view of sensor field used in connection withembodiments of the present invention;

FIG. 4 is a front view of a holographic user interface device accordingto one embodiment of the present invention; and

FIG. 5 is a front view of a holographic user interface device accordingto another embodiment of the present invention.

FIG. 6 is a perspective view of a holographic user interface device andits track of movement in a 3-D coordinate system according to oneembodiment of the present invention.

FIG. 7 is a front view of a computer screen showing an operating systemdesktop in which a holographic mouse can be used to navigate and selectitems quickly and easily.

DETAILED DESCRIPTION OF THE INVENTION

A description of example embodiments of the invention follows.

The teachings of all patents, published applications and referencescited herein are incorporated herein by reference in their entirety.

The present invention, in accordance with one embodiment relates to thecreation of a holographic user interface which transforms the computingenvironment to enable a three dimensional (3-D) holographic style userinterface and display system. The system utilizes holographic projectiontechnology along with programmed quadrant matrixes sensor field tocreate multiple methods to select and interact with data and userinterface tools and icons presented in a holographic format.

Systems related to holographic human machine interfaces have beenpreviously described, e.g., in U.S. Pat. No. 6,031,519 entitled“Holographic Direct Manipulation Interface,” in U.S. Pat. No. 6,377,238entitled “Holographic Control Arrangement,” and in U.S. Pat. No.7,054,045 entitled “Holographic Human-Machine Interfaces,” teachings ofall of which are incorporated herein by reference in their entirety,respectively. For more details on a specific holographic input device, aholographic keyboard displayed on a flat surface is described in greaterdetails in U.S. Patent Application Publication Number 20020070921entitled “Holographic Keyboard,” teachings of which is incorporatedherein by reference in its entirety.

FIG. 1 illustrates a holographic user interface 100 according to oneexample embodiment of the present invention. The holographic userinterface 100 includes a processor 114 that operates software 112,controls a holographic image projector 116, and processes informationobtained from sensors 118 a, 118 b. The projector may generate a 3-Ddisplay image 101, 102 within a 3-D coordinate system 150. The sensors118 a and 118 b may be directed toward the 3-D coordinate system tosense user interaction with images within the 3-D coordinate system. Ifa user were to interact with an image 101 or 102, the sensors 118 a and118 b would provide coordinate information that the processor cancorrelate with the projected images 101 and 102 in the 3-D coordinatesystem.

FIG. 2 is a flow chart that illustrates the method for providing a 3dimensional (3-D) interface with a system. The interface generates (210)an image in a 3-D coordinate system. In operation, an embodiment of theinterface deploys holographic information in the form of a userinterface template as a default once turned on. Sensors on the interfacesense (220) a user's interaction with the 3-D coordinate system. Thesensing may occur through the use of matrixes or triangulated datapoints that correspond to specific functions and data display which thesystem is capable of displaying. The interface may then correlate (230)the user's interaction with an image in the 3-D coordinate system. Bysensing and correlating interaction with the 3-D coordinate system, theinterface allows a computer system or display to interact with a user.The holographic data displayed by the system becomes a result of aselection process by the user who triggers data being displayed by keystrokes or by the use of a three dimensional interactive interface.Users location commands are read by the system at their exact points andthen the system deploys the appropriate response or holographic mediabased upon the users specific request made via the location of thatrequest.

FIG. 3 illustrates a sensor field used in connection with embodiments ofthe present invention. The embodiment illustrated in FIG. 3 includesfour laser sensors 320 a-d. The manipulatable interface may be arelatable and interactive holographic media via the use of a sprocketedsensor system which deploys from the display either via a built in orretrofit hardware peripheral that creates a quadrilateral anglenavigation system to determine the exact point 330 of a fingertip touchpoint 340 within a quadrant 310 (also referred to as a “3-D coordinatesystem”). This touch point, if effectively deployed by the user, ismapped to the image deployed by the holographic hardware and softwaresystem, as each image that is displayed in the system is displayed froman exacting point at an exacting place in space that has beenpreconfigured to match specific points on the quadrilateral sensorsystem. The points in space attached to programmed images are thenmatched to touch points made by the user. The touch point may triggerthe same functions as a mouse and cursor.

One skilled in the art will recognize that other sensing configurationsor devices may be used to sense a location within a 3-D coordinatesystem. For example, the sensors may be laser sensors configured toprovide data to triangulate a point within the 3-D coordinate system,photo voltaic sensors, photo electric light sensors, or image sensors.The sensors may be programmed to identify the specific location of thetouchpoint 330 that may extend through multiple planar images, toidentify a single image located at a 3-D coordinate space.

FIG. 4 illustrates a holographic user interface device 400 according toone embodiment of the present invention. The device 400 has a port 405that may provide the output projector 410 for the multi-dimensionaldisplay, and also the sensors 415 for detecting user interaction. Asillustrated in FIG. 4, a sensor ring 415 may surround a holographicprojector 410. The projector 410 and sensors 415 map out a 3-Dcoordinate system 420 to serve as the holographic user interface. Acommunications port 430, such as a universal serial bus (“USB”) port orwireless connection, serves to allow the device 400 to communicate witha computer system. Alternatively, device 400 may be embedded in acomputer system. The 3-D coordinate system may comprise an image of aninput device 440. As illustrated in FIG. 4 as an example, the 3-Dcoordinate system comprises an image of a typical mouse 440. The sensorscan align with controller options 450 of the mouse in the image andsense interaction with the image by approaching finger(s) and/or hand(s)460. Any system that utilizes holographic displays may also bemanipulated and selected using the sensor interface system.

FIG. 5 illustrates a holographic user interface device 500 according toanother embodiment of the present invention, where an image of a gamecontroller is displayed. The device 500 has a port 505 that may providethe output projector 510 for the multi-dimensional display, and also thesensors 515 for detecting user interaction. As illustrated in FIG. 5, asensor ring 515 may surround a holographic projector 510. The projector510 and sensors 515 map out a 3-D coordinate system 520 to serve as theholographic user interface. A communications port 530, such as auniversal serial bus (“USB”) port or wireless connection, serves toallow the device 500 to communicate with a computer system.Alternatively, device 500 may be embedded in a computer system. The 3-Dcoordinate system may comprise an image of an input device 540. Asillustrated in FIG. 5 as an example, the 3-D coordinate system comprisesan image of a typical game controller 540. The sensors can align withcontroller options 550 of the game controller in the image and senseinteraction with the image by approaching finger(s) and/or hand(s) 560.Any system that utilizes holographic displays may also be manipulatedand selected, using the sensor interface system.

As used herein, an “input device” is a hardware mechanism thattransforms information in the external world for consumption by acomputer. As used herein, a “computer” is any machine which manipulatesdata according to a list of instructions. Often, input devices are underdirect control by a human user, who uses them to communicate commands orother information to be processed by the computer, which may thentransmit feedback to the user through an output device. Input and outputdevices together make up the hardware interface between a computer andthe user or external world. Input devices that can be displayed in the3-D coordinate system according to the present invention include thosethat rely on mechanical motion as the modality of input. Non-limitingexamples of such input devices include: keyboards, mice (many mice allow2-D positional input, but some devices allow 3-D input, such as theLogitech Magellan Space Mouse, manufactured by Logitech of Fremont,Calif.), track ball (some trackballs are integrated into mice), touchpads, SpaceBall (6-degree-of-freedom controller developed by 3DConnexionof San Jose, Calif.), touch screens (touch screens may also function asvisual output devices), graphics tablets (or digitizing tablets) thatuse styluses or fingers to input texts and/or drawings, light pens,light guns, steering wheels, yokes, jog dials, isotonic joysticks (e.g.,analog sticks, where the user can freely change the position of thestick, with more or less constant force), isometric joysticks (e.g.,pointing stick, where the user controls the stick by varying the amountof force they push with, and the position of the stick remains more orless constant), directional pads, wired gloves, ShapeTape (developed byMeasurand Inc of Fredericton, Canada), and other haptic devices. Suchinput devices can be combined on a single physical device that could bethought of as a composite device. Many gaming devices have controllers(game controllers) like this, for example, game pads, game paddles andWii remotes (manufactured by Nintendo of Redmond, Wash.).

User interaction with an image displayed in a 3-D coordinate system cancomprise movement of the user relative to the image. The movement can beeither discrete or continuous. Continuous movements can be eitherunidimensional or multi-dimensional. Accordingly, to sense the userinteraction with, the image, the sensors can comprise any one orcombination of location sensors, motion sensors and light sensors. Thelocation sensors can comprise laser sensors, which can be configured togeometrically identify a position within the 3-D coordinate system,e.g., by triangulation or quadrilation. Non-limiting examples of lightsensors include photovoltaic sensors, image sensors and photo-electriclight sensors.

Continuous movements can either be sensed by a plurality of sensors,each aligned to a fixed space within the 3-D coordinate system, or besensed by one sensor or a plurality of sensors in combination, capableof scanning the entire 3-D coordinate system at a high speed. In thecase of a plurality of sensors each aligned to a fixed space within, the3-D coordinate system, the spaces aligned with the sensors can beoverlapping, contiguous or close enough to each other so that acontinuous movement can be sensed by a plurality of sensors anddigitized into a series of discrete movements at a high-resolutionenough to be thought of continuous. In the case of one sensor or aplurality of sensors in combination, capable of scanning the entire 3-Dcoordinate system at a high speed, the scanning speed can be high enoughso that each change of position of the user above a certain limit(depending on the resolution of the sensor(s)) during a continuousmovement can be captured by the sensor(s) as a different position andthe continuous movement can be digitized into a series of discretepositions at a high-resolution enough to be thought of as a continuousmovement. Sensor technology related to both cases are well known tothose skilled in the art.

User interaction can also comprise any sound produced by the user. Suchsound can be sensed by a sound sensor and correlated by a processor asan audio input from the user.

Certain, input devices typically remain stationary during use. Forexample, a graphics tablet generally remains stationary while a useruses a stylus or a finger to move continuously over it to input textsand/or drawings. In these cases, the continuous input devices arestationary and the user movements (interactions with the continuousinput devices) are continuous. Other input devices move during use, forvarious reasons. Some devices need to move in order for proper inputfunctions to work. For example, mice move with the user's hand in orderto direct a cursor across the screen of a computer monitor. Otherdevices can also move even though movement of these devices does notcontribute to the input function. For example, a game controller can bemade stationary when a user holds it, but it is more desirable for agame controller to be able to move with the user's hands during a gameto provide increased convenience and entertainment for the user.

There are various was to generate a moving image within a 3-D coordinatesystem. For example, a moving image can either be generated by aplurality of projectors, each aligned to a fixed space within the 3-Dcoordinate system, or be generated by one projector or a plurality ofprojectors in combination, capable of changing the position of theprojected image across the entire 3-D coordinate system. In the case ofa plurality of projectors each aligned to a fixed space within the 3-Dcoordinate system, the spaces aligned with the projectors can beoverlapping, contiguous or close enough to each other so that a seriesof overlapping, contiguous or close images can be generated in turn bydifferent projectors. Provided that the distance between two imagesdisplayed consecutively is small enough, the series of images displayedcan be high-resolution enough to be thought of as continuous movement ofthe image. Optionally, one can adjust the duration of time each image isdisplayed to provide a “trailing” effect of the moving image. Forexample, a default situation can be that each image disappears themoment the next consecutive image shows up. However, one can delay thetime before the first image disappears so that the two images canco-exist in the 3-D coordinate system for a given period, of time. Inthis way, the moving image can have a trail behind it when it moveswithin the 3-D coordinate system. In the case of one projector or aplurality of projectors in combination capable of changing the positionof the projected image across the entire 3-D coordinate system, thespeed of changing can be high enough and the position of the changed,image relative to the image preceding it can be small enough (dependingon the resolution of the projectors)) so that a series of consecutivechanges of position of the image can be thought of as a continuousmovement.

The sensors of the present invention can provide information about userinteraction with an image displayed at a location, in a 3-D coordinatesystem. A processor can receive such information, correlate the userinteraction with the image and provide one or more indicationsresponsive to a correlation of the user interaction with the image. Theone or more indications can comprise displaying the image at a secondlocation in the 3-D coordinate system. For example, when the sensing isbased on location and movement of the user relative to the image, one ormore sensors can sense touching of the image of a ball by the hand of auser. A processor can receive such information, correlate the userinteraction with the image and display another image of the ball at asecond location in the 3-D coordinate system opposite to the incomingdirection of the touching hand, thereby creating an effect of the handpushing the ball away. If the sensing is based on sound input, a usercan produce a sound to a sound sensor, e.g., speak to a sound, sensor, aprocessor can correlate this sound input with the image and displayanother image of the ball at a second location within the 3-D coordinatesystem based on the sound input. In the case of voice recognition, e.g.,the user may speak “move up 3 inches,” and the image of the ball, canmove up 3 inches in the 3-D coordinate system.

The image at the first location can cease to be displayed before, whenor after the image at the second location starts to be displayed. In thecase when the image at the first location ceases to be displayed beforethe image at the second location starts to be displayed, there will be aperiod of time during which no such image is displayed in the 3-Dcoordinate system. In the case when the image at the first locationceases to be displayed at exactly the time when the image at the secondlocation starts to be displayed, the images at the two differentlocations do not coexist but there is always one such image in the 3-Dcoordinate system at any given time. In the case when the image at thefirst location ceases to be displayed after the image at the secondlocation starts to be displayed, there will be a period of time duringwhich the image is being displayed at both locations.

The distance between the first location and the second location can bewithin a range such that a sequential display of the image at the firstlocation and the second location can be thought of a movement of theimage from the first location to the second location. As used herein,“sequential” or “consecutive” refers to the fact that the image at thefirst location starts to be displayed earlier than the time the imagestarts to be displayed at the second location, and docs not limit thetime the image ceases to be displayed at the first location relative tothe time the image starts to be displayed at the second location. If thedistance between the two locations of the image is too short, then thechange of the location of display may not be able to be perceived by auser as a movement of the image, i.e., the image may appear stationary.If the distance between the two locations of the image is too long, thenthe display of the image at the second location may be perceived as aseparate event from the display of the image at the first location,rather than a movement of the image from the first location to thesecond, location.

A series of such movements of the image within the 3-D coordinate systemcan be thought of as a continuous movement of the image across the 3-Dcoordinate system. If, for each movement within the series, the imageceases to be displayed at the first location after the image starts tobe displayed at the second, location, by adjusting the time it takesbefore the image ceases to be displayed at the first location after theimage starts to be displayed at the second location, the system of thepresent invention can create a “trailing” effect of a continuousmovement of the image across the 3-D coordinate system.

A movement of an image from a first location to a second location in a3-D coordinate system can be responsive to the movement of the wholebody or one or more body parts of a user from, the first location to thesecond location in the 3-D coordinate system. The image can be displayedat the second location immediately after the whole body or the one ormore body parts of the user moves to the second location, therebycreating an effect of the image moving with the whole body or the one ormore body parts of the user. If the user's movement is continuous, aseries of movements of the image in the 3-D coordinate system can bethought of as a continuous movement of the image across the 3-Dcoordinate system with the user.

The image can be that of an input device for a computer. Non-limitingexamples of such input devices have been described above. According tothe present invention, during a movement of a user across the 3-Dcoordinate system, and a responsive movement of an image of an inputdevice with the user across the 3-D coordinate system, as describedabove, the user can interact with the image at any time during suchmovement. For example, a user can press and hold a button (actuator) onthe image of a mouse while dragging the image of the mouse. Accordingly,one or more sensors can sense the relative position of the user's handto the image of the mouse during its movement and a processor cancorrelate such information (determine if any button/actuator on themouse is being pressed for any period of time during the movement of themouse with the movement of the hand) and input data to a computer towhich the system of the present invention can communicate or of whichthe system of the present invention is a part. For example, the locationsensors can be aligned with each button on the mouse in the imagedisplayed and sense the location of the finger relative to each, button.A processor of the device 400 can receive such user interaction with theimage and correlate the user interaction with the image, e.g., correlatethe locations of the finger relative to each button to actions ofpressing the buttons. For example, if the location of the finger iswithin a certain distance to a button or overlaps with the image of thebutton, then it is considered by the processor that the button is beingpressed. Device 400 can communicate to a computer or can be part of acomputer itself i.e., the processor can operate device 400 or anotherdevice that communicates with device 400 in a manner responsive to acorrelation of the user interaction, with the image. For example, upon,correlating user finger locations to pressing and holding the leftbutton of the mouse as displayed when the hand moves across the 3-Dcoordinate system, device 400 can select and highlight some texts in theapplication Word in a computer, just as pressing and holding the leftbutton of a mouse while dragging it would do. Device 400 does not haveto be a computer by itself with the application Word installed, though,as long as it can provide or cause one or more other devices to providethe appropriate input functions to such a computer, via, for example, aUSB or wireless interface.

In the case of a mouse and other applicable input devices, for whichmovement of the device by itself constitutes a form of input (e.g.,dragging a mouse correlates to movement of a cursor on the screen,moving a Wii remote correlates to a movement in a computer game), themovement of an image of such input devices responsive to the movement ofthe user can result in data input in the absence of any other userinteraction with the devices. In the case where movement of the devicedoes not result in data input but provides convenience and/or enjoymentto the user during use of the device (e.g. a directional pad for videogames), data input into the computer responsive to the user interactionwith the image of the input device can be unrelated to the movement ofthe user across the 3-D coordinate system. For example, user input wheninteracting with an image of a directional pad can result only when theuser presses appropriate buttons in the image of the directional pad,and not when the user holds the image of the directional pad and movesit with the hands. Therefore, the data input to the computer cancomprise data that are related to or not related to the spatialdifference between the first location and the second location in the 3-Dcoordinate system, or both.

Generation of a moving image responsive to user movement as describedabove can be one type of indication provided by a processor, so that themovement of the image can be responsive to the movement of the wholebody or one or more body parts of the user. This can involve certaincomputational algorithms which, after calculation, can provideinstructions to the projector(s) related to the position of a new imageand the time the new image is displayed based on the positional changeof the whole body or one or more body parts of the user within the 3-Dcoordinate system. Methods of developing such computational algorithmsare well known to those skilled in the art.

For example, an image of a mouse can move with the hand holding itacross the 3-D coordinate system. At the same time, the track of themouse, which is the same track as that of the moving hand, which in turncan be determined by one or more sensors, can be determined. A processorcan receive such information (track of the moving hand), correlate itwith the perceived track of the mouse and provide one or moreindications, among which can be moving of a cursor accordingly on acomputer screen.

One issue exists with tracking the movement of an image of an inputdevice in a 3-D coordinate system that is conventionally thought of as adevice moving only two-dimensionally, e.g. a mouse. As illustrated inFIG. 6, the image of a mouse 602 is conventionally thought to move in aplane 601 in a 3-D coordinate system 600. If the image 602 moves to alocation that is not within plane 601, e.g., to location 603 asillustrated in FIG. 2, such movement can be computationally broken downinto two separate movements, one movement within plane 601 to location604, the other vertical to plane 601 to location 605, wherein bothlocations 603 and 605 are within a plane 606 that is parallel to plane601. A processor may determine that only the track from location 602 tolocation 604 within plane 601 contributes to the input, data, e.g.,movement of a cursor on a computer screen. Depending on thecomputational algorithms used and the types of input data, the computercan receive and/or its software can accept, the track from, location 602to location 605 that is vertical to plane 601 may also contribute todata input. This is especially useful for the newest operating systemsthat are capable of simulating a 3-D environment on the desktop in acomputer's monitor. For example, “Windows Flip 3D” developed byMicrosoft Corporation (Redmond, Wash.) and provided in Premium, Businessand Ultimate editions of Windows Vista®, creates a view of open windowsin a 3-D stack on the desktop (FIG. 7). Windows Flip 3D allows a user touse the Windows logo+TAB keys in order to “flip” through the openwindows to quickly locate and select the window one wants. Softwareapplications that can accept data related to the vertical movement orthe vertical element of the movement of an image of a mouse (e.g., thevertical track from location 602 to location 605, broken down from themovement of the mouse from location 602 to location 603, as illustratedin FIG. 6) in a 3-D coordinate system according to the present inventioncan allow a user to directly locate the window one wants without havingto flip through the stack of windows. Another exemplary situation wherethe vertical movement or the vertical element of the movement of animage of a mouse can contribute to data input is in the realm ofcomputer-assisted design, where movement of a mouse in a 3-D coordinatesystem can directly result in input of spatial data in designing 3-Dstructures.

User interaction with the image of an input device can also comprise anysound produced by the user. Such sound can be sensed by a sound sensorand correlated by a processor as an audio input to, e.g., device 400.Either device 400 or another device to which device 400 can communicatecan convert and/or transmit the audio input. For example, an audio inputcan be in the form of a voice command, wherein the processor cancorrelate the voice command (voice recognition) and operate device 400or a device that communicates with it in a manner as dictated by thevoice command. The operation dictated by the voice command, however, ispreferably related to a function of the input device as displayed in theimage, e.g., a user may use voice command to drag a mouse if the inputdevice displayed in the image is a mouse. This can provide greatconvenience for people with impaired, mobility. By providing voicecommand, the user can visualize the movement and/or clicking of themouse across the 3-D coordinate system without physically interactingwith the image, while at the same time, the movement and/or clicking ofthe mouse image correlates with corresponding data input into thecomputer.

The 3-D image of the input device according to the present inventionessentially becomes a virtual input device for any system that requiressuch input device, especially when the input device typically movesduring use. The 3-D image of the input device can be used together withother input devices or images of other input devices simultaneously,either in association or independently, in the same system. For example,the system can also display a holographic image of a keyboard in thesame 3-D coordinate system in which a holographic mouse is displayed.The holographic keyboard can serve as a virtual keyboard and theholographic mouse can serve as a virtual mouse, both contributing todata input to a computer. The holographic keyboard is further explainedin details in U.S. patent application Ser. No. 11/932,731 entitled,“Digital, data, and multimedia user interface with a keyboard” by GeneS. Fein and Edward Merritt, the teaching of which is incorporated hereinby reference in its entirety. Alternatively, the 3-D image of a mousecan be the sole virtual input device for a computer, while the computercan still have an actual keyboard. Therefore, the virtual input devicesof the present invention can either work independently, or in tandemwith any other virtual or actual input devices. There are numerousadvantages for having a virtual input device in addition to the onesdescribed above. Sometimes the actual devices are too big to carry andaffect mobility of the users needing them. Instead of using the actualdevice, those people can interact with the 3-D image of the devicegenerated by a much smaller device on the go without having to worryabout where to store the actual, bigger device. Sometimes people withvisual impairment or other disabilities may find certain input devicestoo small to operate. The system according to the present invention canprovide a 3-D image of the continuous input device much larger than theactual device. Instead of using the actual device, those people caninteract with, the 3-D image of the device much more easily.

Those of ordinary skill in the art should recognize that methodsinvolved in providing a 3-D user interface with a system may be embodiedin a computer program product that includes a computer usable medium.For example, such a computer usable medium can include a readable memorydevice, such as a solid state memory device, a hard drive device, aCD-ROM, a DVD-ROM, or a computer diskette, having storedcomputer-readable program code segments. The computer readable mediumcan also include a communications or transmission medium, such aselectromagnetic signals propagating on a computer network, a bus or acommunications link, either optical, wired, or wireless, carryingprogram code segments as digital or analog data signals. The programcode enables and supports computer implementation of the operationsdescribed in FIGS. 1 and 2 or other embodiments.

While this invention has been particularly shown and described withreferences to example embodiments thereof, it will be understood bythose skilled in the art that various changes in form and details may bemade therein without departing from the scope of the inventionencompassed by the appended claims.

What is claimed is:
 1. A system for a 3 dimensional (3-D) userinterface, the system comprising: one or more 3-D projectors configuredto display a holographic image at a first location in a certain 3-Dcoordinate system; multiple sensors configured to sense, based onquadrilateral angle navigation to determine a touch point positionwithin the certain 3-D coordinate system, user interaction with theholographic image within the certain 3-D coordinate system and toprovide user interaction information; and a processor configured to: (i)receive the user interaction information from the multiple sensors; (ii)based on the received user interaction information, correlate the senseduser interaction with the holographic image using the certain 3-Dcoordinate system; and (iii) provide one or more indications responsiveto a correlation of the user sensed interaction with the holographicimage, wherein the one or more indications comprise displaying theholographic image at a second location in the 3-D coordinate system. 2.The system of claim 1, further comprising a communications port coupledto the processor and configured to provide communications interface witha computer system.
 3. The system of claim 1, wherein the userinteraction with the image comprises movement of the user relative tothe image, or any sound produced by the user, or both.
 4. The system ofclaim 1, wherein the multiple sensors comprise any one or combination oflocation sensors, motion sensors, light sensors and sound sensors. 5.The system of claim 4, wherein the location sensors comprise lasersensors configured to geometrically identify a position within the 3-Dcoordinate system.
 6. The system of claim 1, wherein the image at thefirst location ceases to be displayed before, when or after the image atthe second location starts to be displayed.
 7. The system of claim 1,wherein the distance between the first location and the second locationare within a range such that a sequential display of the image at thefirst location and the second location can be thought of as a movementof the image from the first location to the second location.
 8. Thesystem of claim 7, wherein the user interaction comprises movement ofthe whole body or one or more body parts of the user from the firstlocation to the second location, thereby creating an effect of the imagemoving from the first location to the second location responsive to themovement of the whole body or the one or more body parts of the userfrom the first location to the second location.
 9. The system of claim8, wherein the image is displayed at the second location immediatelyafter the whole body or the one or more body parts of the user moves tothe second location, thereby creating an effect of the image moving withthe whole body or the one or more body parts of the user.
 10. The systemof claim 9, wherein the image is that of an input device for a computer,and the one or more indications further comprise inputting data to thecomputer responsive to the correlation of the user interaction with theimage.
 11. The system of claim 10, wherein the data input to thecomputer comprise data that are related or not related to the spatialdifference between the first location and the second location in the 3-Dcoordinate system, or both.
 12. The system of claim 10, wherein theinput device is any one or combination of: a mouse, track balls, lightguns, steering wheels, yokes, joysticks, directional pads, wired gloves,game controllers, game pads, game paddles and Wii remotes.
 13. Thesystem of claim 10, wherein the image of the input device is any one orcombination of a keyboard and a mouse, and the one or more indicationsfurther comprise input to the computer through one of a keyboard and amouse.
 14. A method for providing a 3 dimensional (3-D) user interface,the method comprising: generating a holographic image at a firstlocation in a certain 3-D coordinate system; sensing, based onquadrilateral angle navigation to determine a touch point position withthe certain 3-D coordinate system, user interaction with the holographicimage within the 3-D coordinate system; correlating the sensed userinteraction with the holographic image using the 3-D coordinate system;and providing one or more indications responsive to a correlation of thesensed user interaction with the holographic image, wherein the one ormore indications comprise displaying the holographic image at a secondlocation in the 3-D coordinate system.
 15. The method of claim 14,wherein the user interaction with the holographic image comprisesmovement of the user relative to the holographic image, or any soundproduced by the user, or both.
 16. The method of claim 14, whereinsensing comprises using one or more laser sensors to geometricallyidentify a position within the 3-D coordinate system.
 17. The method ofclaim 14, wherein sensing comprises using a sound sensor to identify anaudio input from the user.
 18. The method of claim 14, wherein theholographic image at the first location ceases to be displayed before,when or after the holographic image at the second location starts to bedisplayed.
 19. The method of claim 14, wherein the distance between thefirst location and the second location are within a range such that asequential display of the holographic image at the first location andthe second location can be thought of as a movement of the holographicimage from the first location to the second location.
 20. The method ofclaim 19, wherein the user interaction comprises movement of the wholebody or one or more body parts of the user from the first location tothe second location, thereby creating an effect of the image moving fromthe first location to the second location responsive to the movement ofthe whole body or the one or more body parts of the user from the firstlocation to the second location.
 21. The method of claim 20, wherein theholographic image is displayed at the second location immediately afterthe whole body or the one or more body parts of the user moves to thesecond location, thereby creating an effect of the holographic imagemoving with the whole body or one or more body parts of the user. 22.The method of claim 21, wherein the holographic image is that of aninput device for a computer, and the one or more indications furthercomprise inputting data to the computer responsive to the correlation ofthe user interaction with the image.
 23. A computer program productcomprising: a non-transitory computer readable storage medium havingcomputer readable program codes embodied therein for causing a computerto provide a 3 dimensional (3-D) user interface, the computer readableprogram codes causing the computer to: generate a holographic image at afirst location in a certain 3-D coordinate system; sense, based onquadrilateral angle navigation to determine a touch point positionwithin the certain 3-D coordinate system, user interaction with theholographic image within the 3-D coordinate system; correlate the senseduser interaction with the holographic image using the 3-D coordinatesystem; and provide one or more indications responsive to a correlationof the sensed user interaction with the holographic image, wherein theone or more indications comprise generating the holographic image at asecond location in the 3-D coordinate system.